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CONVEX  DUALITY  APPROACH 
TO  THE  OPTIMAL  CONTROL  OF  DIFFUSIONS 

Wendell  H.  Fleming f  and  Domokos  Vermes* 


1.  Introduction 


\\e  consider  R 
tion 


'  ci  l  UE  ci 


dxs  =  b(stxs.ua)ds  +  a(s,xa,us)du's,  xt  =  x  (1.1  i 

with  wa  an  R"- valued  Brownian  motion  and  ua  a  non-anticipative  Y  CC  Rn -valued  control 
process.  The  objective  is  to  minimize  the  expected  (possibly  discounted)  cost 


rT 

Jv(t,x)-.=  E?,x  J  e-c(i'r'’,")/(s,ii,us)d5 


(1.2) 


over  all  control  processes  u.  Here  T  is  a  finite  or  infinite  planning  horizon.  Additional 
terminal  costs  could  also  be  included. 

An  important  feature  of  the  present  paper  is  that  we  do  not  make  any  ellipticity  assump¬ 
tion,  the  matrix  a  can  be  degenerate  or  even  identically  zero.  This  means  the  approach 
covers  both  deterministic  and  stochastic  control  theory. 

Another  specialty  is  that  the  running  cost  (and  terminal  cost  if  present)  is  not  required 
to  be  bounded  or  continuous,  merely  lower  semi- continuous  and  of  polynomial  growth. 
This  makes  it  possible,  among  other  things  to  include  also  problems  where  the  objective 
is  e.g.  to  minimize  the  probability  of  the  event  that  the  state  ever  leaves  a  closed  subset 
of  the  state  space  or  to  maximize  the  hitting  probability  of  a  target  set;  and  in  particular 
to  cover  the  fixed  end-point  problem  of  deterministic  control  theory. 

In  distinction  from  most  papers  in  the  field,  the  present  approach  does  not  use  dynamic 
programming  but  is  based  on  duality  of  convex  analysis.  We  embed  our  control  problem 
into  a  convex  mathematical  programming  problem  on  a  space  of  measures  and  consider 
its  dual  which  turns  out  to  involve  the  Hamilton- Jacobi- Bellman  (HJB)  equation.  More 
precisely  we  find  that  the  dual  of  the  original  minimization  problem  is  to  seek  the  supremum 
of  all  smooth  subsolutions  of  the  Hamilton-Jacobi-Bellman  equation.  From  the  existence 
of  an  equilibrium  point  for  the  primal-dual  game  it  then  follows,  in  particular,  that  the 
optimal  value  function  is  the  upper  envelope  of  the  smooth  subsolutions  of  the  Hamilton- 
Jacobi-Bellman  equation. 
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The  proof  consists  of  two  major  steps.  First  we  construct  the  minimization  problem 
on  the  space  of  measures  which  contains  the  original  control  problem  embedded  (§3  i  and 
apply  the  Fenchel-Rockafellar  duality  theorem  [4]  to  arrive  at  the  HJB  equation  ^ d, 4 ) .  In 
the  second  step  we  prove  that  the  embedding  is  actually  tight;  the  infimum  is  the  sr.n.r 
both  in  the  original  and  in  the  extended  problem  (§§5-6).  This  second  part  of  the  proof 
is  based  on  the  separation  theorem  and  uses  some  analytic  tools  like  mollification  and 
Sobolev  estimates,  which  in  turn  are  derived  by  control-theoretic  arguments.  Roughly  one 
could  say  that  the  separation  is  carried  out  by  a  sufficiently  smooth  control  problem. 

The  usefulness  of  the  duality  theorem  in  control  theory  was  first  demonstrated  by  Vinter 
and  Lewis  [6],  [7]  who  proved  similar  results  for  deterministic  control  problems.  Theii 
approach  was  made  available  for  stochastic  control  problems  in  [5]  by  basing  it  on  the 
theory  of  occupation  (potential  and  harmonic)  measures  and  infinitesimal  operators.  The 
present  paper  extends  the  method  to  the  optimal  control  of  diffusions.  Since  the  diffusion 
matrix  is  allowed  to  degenerate,  the  presented  results  apply  uniformly  to  both  deterministic 
and  stochastic  control  problems.  The  novel  proof  of  the  tightness  of  the  embedding  is  not 
only  more  general  but  even  in  the  classical  deterministic  case  it  is  more  direct  than  the 
arguments  of  [G]. 

In  [3]  Lions  characterizes  the  optimal  value  function  of  stochastic  control  as  the  largest 
generalized  subsolution  of  the  Hamilton-Jacobi-Bellman  equation.  The  approach  and 
method  of  proof  differs  from  the  one  followed  here. 


2.  Formulation  of  the  problem 

Let  T  be  the  planning  horizon,  either  a  non-negative  number  or  +  oo.  We  take  0  < 
t  <  T.  If  T  <  oc  then  the  state  space  will  be  E°  :=  [0,T]  x  R"  and  if  T  =  -foo  then 
E°  :=  [0,T)  x  R".  We  denote  by  E  the  one-point  compactification  of  E°  and  introduce 
the  notation  5°  ;=  E°  x  Y  and  5  :=  E  x  Y.  Note  that  E  and  S  are  compact. 

The  coefficients  a(t,x,y)  and  b(t,x,y)  as  well  as  the  discount  rate  c(f,x,y)  >  0  are 
assumed  to  be  bounded  continuous  functions  on  S°  such  that  their  first  partial  derivatives 
with  respect  to  t  and  second  partial  derivatives  with  respect  to  x  exist  and,  together  with 
the  functions  themselves  can  continuously  be  extended  to  5.  The  running  cost  /  is  assumed 
to  be  lower  semi-continuous  on  5  and  of  at  most  polynomial  growth.  The  case  of  additional 
terminal  costs  will  be  considered  in  §8. 

For  simplicity  we  assume  that  either  the  planning  horizon  T  is  finite  or  that  there  is 
a  strict  discounting,  i.e.  co  =  inf^gs0  c(cr)  >  0-  The  effect  of  the  discounting  will  be 
included  into  the  process  as  an  exponential  killing  or  a  jump  to  the  fictitious  isolated 
cemetery  state  A  at  the  killing  time  0.  In  what  follows  all  expectation  signs  E  will  refer 
to  the  killed  process.  The  only  exception  is  the  sans  serif  E  in  formula  (1.2)  which  denotes 
the  expectation  of  the  non-killed  process,  i.e. 

E$(xr)  =  £$(xr)  •  l{©>r}  =  E$(zr)  e~c(i,x,,Ut)  ds. 

We  will  also  use  the  notation  r  :=  min (0.T)  and  refer  to  it  as  the  life-time  of  the 
processes.  The  cost  Ju  can  then  be  expressed  in  the  three  equivalent  forms 


(1.2') 


Jv(1.x) 


=  E“r  J*  e-cl9-x‘'u‘)l(s.x„ut)ds 
=  E?,x  f  l(s,xs,us)ds  =  Elz  £  l(s, 


Xs, 


ut )  ds. 


The  assumptions  about  the  boundedness  of  the  coefficients,  growth  of  the  costs,  and 
boundedness  of  the  expected  life-time  can  be  substantially  relaxed.  In  fact,  the  proofs  use 
a  much  less  stringent  but  also  less  explicit  assumption;  c.f.  the  remark  following  Lemma 
2.1. 

The  spaces  of  functions  on  S°  and  E°  which  sire  continuously  extendable  to  S  and  E 
will  be  denoted  by  C(S)  and  C(E)  respectively  and  they  are  considered  to  be  Banach 
spaces  nonned  by  the  supremum  norm.  In  Lemma  2.1  we  will  introduce  a  continuous 
positive  weight  function  7  :  [0,  T)  X  Rn  — »  (0,  00)  associated  with  the  control  problem 
under  investigation.  We  will  consider  the  weighted  spaces 

C-.(S)  :=  {/  €  C(S°)  :  fh  G  C(S),  ||/||,  :=  sup  \f{t,v)\hti)  <  oc 

(er.yey 

and  lim  \f((,  y)|/7’(0  =  0}, 

|«|-.oc 

C\(E)  is  defined  analogously. 

C*{E)  :=  G  C,(E)  :  *(T,z)/y(T,x)  =  0,  €  C,(E)  V  ij  =  1,.. .  ,n}. 

In  the  subsequent  expositions  C*  can  always  be  substituted  by  the  set  of  all  infinitely 
often  differentiable  functions  satisfying  the  boundary  condition  3>(T,  x)/ 7(T,  x)  =  0  and 
with  all  derivatives  in  C1(E).  We  will  refer  to  the  elements  of  C %  as  smooth  functions. 

will  denote  the  space  of  all  signed  Borel  measures  M  on  5°  for  which  the  norm 
||M||..  =  f  7  dM+  +  f  7  dM~  is  finite.  Here  M+  and  M~  are  the  positive  and  negative 
parts  of  the  Jordan  decomposition  of  M.  With  obvious  identification  elements  of 
can  be  considered  as  signed  measures  on  S  not  assigning  mass  to  {00}  x  Y. 

If  T  is  a  positive  constant  then  A/f7,r(S)  will  denote  those  non-negative  measures  from 
M ±15)  for  which  |).1/||^  <  T  <  -foe. 

The  set  U  of  all  admissible  controls  consists  of  all  y-valued  control  processes  us  which  are 
progressively  measurable  with  respect  to  the  filtration  of  the  Brownian  motion  w„.  If  u  G  U 
then  x"  denotes  the  solution  of  the  stochastic  differential  equation  (1.1)  corresponding  to 
it,  satisfying  the  initial  condition  x“  =  x  and  killed  at  rate  c(  ).  The  corresponding 
expectation  operator  will  be  denoted  by  E\ x  and  if  no  confusion  can  arise  the  superscript 
u  will  be  omitted  from  x"  inside  the  expectation. 

With  each  contxol  11  G  U  we  associate  the  measure  Mv  defined  on  the  compact  space 
5  =  £xl"  which  is  the  extension  of 

3 


M  (Di  x  B  x  x  By  )  Et  z  I  lsi(x^)-lQi[us)ds 

M(oc  x  )')  :=  0.  (2.1  i 

Here  Bt  C  [0,oo],  Bx  C  Rn,  By  C  Y  are  arbitrary  Borel  sets  and  1b  denotes  the 
indicator  function  of  the  set  B.  Note  that  though  the  notation  does  not  indicate  it.  the 
measures  Mu  depend  on  the  initial  condition  xt  =  x  in  (1.1)  which  is  considered  to  be 
fixed.  We  will  denote  the  set  of  all  such  A/u  corresponding  to  some  u  £  U  by  A4s(t,  x). 

Intuitively,  il/([t.f']  x  Bx  x  By )  measures  the  expected  time  before  t'  spent  by  the  killed 
process  x"  in  the  set  Bx  while  control  values  from  By  C  Y  were  supplied.  In  particular. 
•,  Y )  is  the  potential  (or  occupation)  measure  of  the  killed  time-space  process  (5,  x” ). 
The  infinitesimal  operator  of  the  killed  Markov  process  x*'  corresponding  to  the  constant 
control  ut  =  y  £  Y  is  defined  for  each  $  6  C2(E°)  and  is  given  by  the  expression 

1,1=1  ‘  J  1=1  ' 

with  ((ljj)  =  ~crT  ■  a.  We  will  use  this  notation  also  for  non-smooth  functions  i.e.  to 
denote  the  value  of  the  expression  on  the  right-hand  side  at  every  point  (t,x)  where  the 
corresponding  partial  derivatives  exist. 

To  interconnect  the  assumptions  on  discounting,  termination  and  growth  as  well  as  to 
express  them  in  a  technically  convenient  analytic  form  we  prove  the  following 

LEMMA  2.1.  There  exist  constants  0  <  a  <  d  and  a  twice  continuously  differentiable 
function  7  :  (O.T)  x  Rn  — >  (0,oo)  satisfying 

0  <  a 7  <  —Av 7  <  07  (2.2) 

everywhere  in  (0,  T)  x  Rn  for  all  y  £  Y. 

PROOF:  We  will  construct  7  separately  for  the  discounted  and  for  the  finite  horizon  case. 

1.)  Discounted  case.  The  infinitesimal  operator  of  the  exponentially  killed  process  is  of 
the  form  A<b  —  DQ  —  c<£  with  D  a  (possibly  degenerate)  second  order  differential  operator. 
We  define 

n 

7(t,x)  :=  (cosh pt)JJ cosh pxi  (2.3) 


with  a  p  yet  to  be  determined  approximately. 

A  straightforward  calculation  shows  that 

-A'(pb(t.x)  <  Dy(t,x)  <  K{p)y(t.x) 
with  K(p)  =  53 "=1  |ja ij  ||p2  +  53"=i  IIMI P  =  ap2  +  bp.  Consequently 


^  “  »f  * » ^ ^  j 


s 


i 


A'(;>)  +  |k||  •  7  <  .47  =  Dy  -  c  •  7  <  (I\(p)  -  c)  ■  7.  (2.5 1 

If  Co  =  inf  c  >  0  then  the  quadratic  equation  c/c  +  fcp  —  c0  =  0  has  exactly  one  positivi 
root  p0-  Choosing  p  from  the  interval  (0,po)  we  get  that  c0  —  K(p)  >  0.  hence  (2.2)  h 
satisfied  with  a  :=  c0  —  A'(p)  and  6  :=  j|c|j  -f  K(p). 

2.  Finite  horizon  case.  We  define 

n 

7(/.  x)  :=  [1  +  {T  -  t)}  ■  coshpx,  =  [1  +  (T  -  Obo(x). 

1=1 

Using  the  notation  .4$  =  d^/dt  +  Dz$  a  calculation  analogous  to  that  of  the  discounted 
case  yields 


—A-)\t,x)  -  ^0{x)-{\  +  T -t)Dz~,0(x)  >[(1+T-1)  1  -K(p)\  ■  {\ +  T -t)  ■  y0(x).  (2.0) 

With  (1  +  T)_1  in  place  of  Co,  the  above  argument  shows  that  if  p  is  chosen  from  (O.pu) 
then  7  satisfies  (2.2)  with  a  :=  (1  +  T)-1  —  K(p)  >  0  and  d  :=  1  +  K(p).  The  proof  of  the 
lemma  is  complete. 

We  formulate  some  consequences  of  Lemma  2.1  which  will  be  used  at  various  places 
during  the  subsequent  expositions. 

Corollary. 

(1)  f  7  dM'1  <  7(f,x)/a  <  -foe  for  every  A/u  €  MS(t,x).  In  other  words,  the  constant 
r  :=  7(/.t)/o  <  -foe  is  a  uniform  upper  bound  for  the  expressions  E*z  f(  7 (s.x^)ds  for 
every  process  .2  “  generated  by  a  control  u  €  U  and  starting  from  initial  state  xt  =  x. 

(2)  7 {t.x)  grows  asymptotically  not  faster  than  an  exponential  function  as  \x\  — +  oc. 
t  — >  cv 

(3)  For  every  (t,  x)  G  E°,  0  <  s  <  +00  and  u  G  U  we  have 

1  -  e~os  <  1  -  '>~1(t,x)Etu<zy(t  +  s,x<+3)  <  1.  (2.7) 

Proof-. 

(1)  follows  from  Dynkin's  formula.  In  fact,  if  T  <  00  we  have 

f  idU"  <  I  J(-A^)dMu  =  j\-Au7)(s,x^)ds 

=  x)  -  7(T,xr)]  <  7(f , x )/q. 

a 

Since  the  botmd  is  independent  of  T.  the  inequality  remains  true  as  T  — >  -foe. 

(2)  is  immediate  from  the  construction  of  7  in  the  proof  of  Lemma  2.1. 

(3)  The  left-hand  side  of  (2.2)  can  be  written  as  .4l*y  -f  ay  <  0.  By  the  Feynman  -  Kac 
formula  it  follows  that  E*xea3't(t  +  s,x“+J)  <  7 (t,x)  with  an  a  >  0.  Subtracting  both 


Xk>  .>  w>  ;> 


sides  of  inequality  £’"T7(i  +  s,  x“+#)  <  e  as~)(i.  r)  from  -)(/,  x)  gives  7  —  £7  >  -.fl  —  < 
which  proves  the  left-hand  side  of  (2.7).  The  right-hand  side  is  trivial  since  *  _1E-  >  0. 

Remark.  The  growth,  discounting  and  termination  conditions  required  earlier  in  tin- 
section  will  be  used  in  the  subsequent  expositions  only  indirectly  through  the  statement 
of  Lemma  2.1.  Consequently  all  results  of  this  paper  remain  valid  under  other  set.-  o: 
assumptions  which  assure  the  existence  of  a  7  with  property  (2.2).  Examples  of  othe: 
possible  sets  of  such  assumptions  are 

(i)  Coefficients  a^.b,  satisfy  linear  growth  conditions,  the  discounting  is  strict,  the  run¬ 
ning  cost  is  bounded.  In  this  case  7  can  be  chosen  asymptotically  as  (a*  I7*  with  p  <  Co  and 
Q  =  Co  -  p. 

(ii)  Coefficients  atj,b,  satisfy  linear  growth  conditions,  the  time  horizon  is  finite,  the 
running  cost  is  of  polynomial  growth.  Then  one  can  choose  7 (t,x  )~[l  +  K{T-t))\x\'1 
with  an  appropriate  I\  and  p. 

Now  we  return  to  our  original  control  problem.  Although  we  assumed  /  to  be  only  lov.vi 
semi-continuous,  in  §§3-G  of  the  paper  we  will  consider  continuous  running  costs.  The 
extension  of  all  obtained  results  to  the  general  semi- continuous  case  will  be  an  additional 
step  in  §7.  With  the  notation  introduced  the  control  problem  we  will  consider  is  §§3-G  can 
be  formulated  ns  the 


Strong  Problem.  For  a  given  running  cost  l  £  Cy(E)  and  initial  state  (f,r)  6  £° 


minimize  j  ldMu  overall  Mv  g  MS(t.x). 

We  can  define  the  optimal  value  V’  of  the  strong  problem  as  a  function  of  the  initial  state 

tlit,x)  :=  inf {  J  ldMu  :  Mu  g  AlS(Cx)}. 


3.  The  weak  formulation  of  the  control  problem 

It  follows  from  Ito’s  formula,  that  for  arbitrary  non-anticipative  control  process  11  g  U 
the  generalization  of  the  fundamental  theorem  of  calculus  (Dynkin’s  formula)  holds  true. 
For  every  twice  continuously  differentiable  $  we  have 

Elx*{a,x„)  -  S(/,r)  =  E «x  £  Au-*(s,xa)ds  (3.1) 

provided  a  <  r  is  a  stopping  time  such  that  the  expectations  exist. 

If  we  apply  this  formula  to  the  terminal  time  r  and  to  smooth  functions  $  g  C’*  which 
vanish  at  the  terminal  state  A  then  by  $(r, xr)  =  $(A)  =  0  we  find  that 


(3.2 


:: 

4 

i 


I 

I 


-Q(t.x)  =  j  Ay$(t',x')Mv(dl'.dx'. 


holds  true  for  every  u  G  U  whenever  ^44“  G  C’v 
We  introduce  the  notations 


MA(t,x):=  {M  G  Ml(S):-$(1.x)  =  J  dM  for  all  $G^(£)) 


M U  (t,x)  :=  M'"r(S)  n  MA(t,x)  with  r  =  j(t,x)/a. 

Since  for  every  u  G  Af  the  measure  Mu  G  A4s(t,x)  is  in  both  A4"|,,r(5)  and 
our  original  control  problem,  the  “Strong  Problem"  is  embedded  in  the  following 


Weak  Problem 


Minimize 


over  M  G  ,M  *  (f,x) 


This  is  a  minimization  problem  on  the  space  of  measures  with  hnear  objective  and 
convex  constraints  In  fact  if  /  G  C-,(S)  then  by  Riesz’  theorem  f  I  dM  is  a  continuous 
linear  functional  on  the  space  of  signed  measures  For  each  $  G  relation  (3.2) 

imposes  a  continuous  linear  restriction  on  M.  consequently  their  intersection  ,Vf..j(/.x)  is 
a  closed  linear  set  in  M±.  Finally  M ~,'r  is  a  uWcompact  convex  subset  of  M\. 

The  feasible  set  of  the  strong  problem  consists  of  all  Mu  G  MS  generated  by  a  control 
u  G  U  via  the  stochastic  differential  equation  (1.1).  This  set  is  contained  in  the  feasible  set 
MW  of  the  weak  problem,  thus  the  optimal  value  t/>(f,x)  :=  inf{/ ldMu  :  u  G  U)  is  not 
less  than  the  minimum  'P(t,x)  =  inf{/ ldM;M  G  ,M”  (<,x)}  in  the  weak  problem.  Note 
that  the  initial  state  (f,x)  is  involved  in  the  strong  problem  through  the  initial  condition 
(1.2)  and  in  the  weak  problem  through  the  definition  of  AIa^.x). 

In  what  follows,  we  will  first  characterize  the  value  function  ^(t,x)  of  the  weak  problem 
by  solving  its  dual,  a  maximization  problem  in  the  function  space  C^E)  C  C7(5).  More 
precisely  it  will  turn  out  that  the  dual  of  the  minimization  problem  (3.3)  is  to  find  the 
supremum  of  all  smooth  subsolutions  to  the  Hamilton- Jacobi  equation. 

To  make  duality  methods  applicable  it  is  convenient  to  bring  the  weak  problem  to 
the  Fenchel  normalform.  Using  extended  valued  functions  wTe  reformulate  the  convcxly 
constrained  linear  problem  as  an  unconstrained  convex  problem.  In  fact,  we  introduce  the 
functionals  /i]  and  h2  :  .Mj.CS)  — ►  R  by 


f  o 

h2{M)  :=(_ 


JldM  if  M  G  A4'r,r(S) 
-foe  otherwise 

0  if  M  G  MA(i,  x) 

—  oo  otherwise. 


Both  /( j  and  —  h2  are  convex  and  lower  semi-continuous.  It  is  immediate  that  the  weak 
problem  is  equivalent  to  the  following 


Fonchol  Problem  Minimize  h\(M)  —  li2(M  )  over  all  M  G  *. 


1 

m 


% 

* 


;»S 

'ft 

S 


I 
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4.  Duality  and  the  Hamilton-Jacobi  problem 

Recall  that  the  space  S  is  compact,  thus  by  Riesz"  theorem  C*(S)  =  A4±(*S)-  In  other 
words  C7(5j  and  ,Vf^.(5j  are  spaces  in  duality  connected  by  the  bilinear  form. 


(<*■!')  =  I 


<t>  G  C-j,  )J,  G 


The  norm  topology  of  C-,  and  the  weak*-  topology  of  M.  J  are  compatible  with  the  pairing, 
the  continuous  linear  functionals  on  both  spaces  are  exactly  those  representable  by  the 
bilinear  form.  If  H  and  h  are  convex'  real-valued  functions  defined  on  C7(S)  and  Ai±(S) 
respectively  then  their  Legendre-Fenchel  transforms  (convex  conjugates)  are  defined  by 


:  <?  G  C7(S)} 

(4.2) 

/rG  AfJ(S)}. 

(4.3) 

If  the  original  function  h  or  H  was  convex  and  lower  semi-continuous  then  it  coincides 
with  its  double  conjugate,  i.e.  H **  =  H ,  /i**  =  h.  Conjugates  of  concave  functions  are 
defined  analogously  but  with  inf  in  place  of  sup,  and  have  the  corresponding  properties. 

Now  we  compute  the  Legendre-Fenchel  transforms  of  the  functionals  h i  and  h 2.  We  use 
the  quantities  '  and  a  as  they  were  introduced  in  Lemma  2.1. 

Lemma  (4.1).  h\(d>)  =  a-1  •  y(t,x)  •  ||(<£  -  /)+||7  =  a-1  •  y(t,x) 

•  sup { [<3(  c)  —  l(a)\/~f(6,  £)  :  over  all  o  =  (6.  £,  77)  G  5  such  that  &(cr)  —  1(a)  >  0}.j 

Proof: 

h\(<t>)  =  supj  j  -  hi(v)  :  ft  G  A4±}  =  supj  J \<f>  —  l)dM  :  M  G  A47,r| 

=  suj)|y>[(0  —  l)/')]')  dM  :  M  >  0,  J  dM  <  T  =  y(t, x)/o  } .  (4.4) 

Since  6  and  /  arc  in  C7,  the  continuous  function  (<f>—l)/y  attains  its  maximum  at  some  point 
Go  =  (to,  s'Oi  yo)  of  the  compact  set  S.  If  (<f>  —  l)(to,  Xo,  yo)/7(to,  x0)  >  0  then  f0  <  00,  x0  ^ 
00  and  the  sup  in  (3.4)  can  be  attained  by  concentrating  all  available  mass  of  the  measure 
y  dM  to  the  point  g0  G  S.  We  have  to  choose  M(  ds)  :=  y(t,x)/(a  ■  y(to.  x0))6ao  (ds)  with 
Sao  denoting  the  Dirac  measure  assigning  unit  mass  to  the  singleton  {eg}.  Then  we  have 

t(/)+  denotes  the  positive  part  of  the  function  /,  i.e.  f+(x)  =  max{0,/(i)} 
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h*(6)  =  .  x)(o  -  /)(<70)/(q  ■  ->(/0,To))  =  a  1  ■  7(L*)||(d  -  /) |U  (4.5' 

provided  sup(b  —  /)  >0. 

If  sup(o  —  /)  <  0.  i.e.  if  d(cr)  <  /(<r)  for  all  cr  €  5,  then  the  maximum  of  the  expression 
(4.4)  is  zero  and  is  attained  for  M  =  0.  This  together  with  (4.5)  proves  the  lemma. 


Lemma  (4.2). 


l2 C^)  “  |  _ 


—  lim  $,(/.  r)  if  d  =  lim,. 
— oo  otherwise. 


.4$,  with  G  C^(£), 


PROOF:  Since  7/ 2  is  concave,  h^io)  :  =  inf {/ (pdAl  —  /?2(M)  :  M  G  A4^(f.a")). 

Let  us  first  assume  that  <f>  =  ,A$  G  C7  with  some  $  G  C^.  Then,  by  the  definition 
of  M^it.  x)  for  every  M  G  Ma{U^)  we  have  f  d  dM  =  f  A$  dM  =  —  $(/,x).  Since 
Ma  is  non-empty.  .4$!  =  .4$2  implies  $j(t,x)  =  $2(f,x)  and  hence  we  have  f  <p  dM  — 
f  .4$  dM  —  —4* ff.r )  whenever  d  =  A$  G  C-,  with  some  $  G  Cy 

Let  us  assume  now  that  there  exists  a  sequence  $k  G  such  that  ||<?  —  — *•  0. 

This  means  that  A$k  /  7  — »  d/7  uniformly  on  S  as  I  -+  00.  By  this  uniform  convergence 
and  the  finitrnes-  of  the  measure  -)  dA/  for  every  A/  G  A4yi(f,x)  C  we  have 

f  cpdM  =  /  —  •  7  dM  =  [  lim  ^--7 dM  —  f  lim  A$k  dM  =\\m$k(t,x) 

J  J  7  J  k~oc  7  d  00 

independently  of  the  particular  choice  of  the  sequence  .4$*.  Since  —  lim$*(f.;r)  does  not 
depend  on  M  G  Ma( L  .t).  we  have  proved  the  first  line  of  (4.6). 

It  remains  to  show  that  ft2(d)  =  —  00  if  d  is  not  in  the  ||  ||7  closure  of  the  functions 
.4$  with  4’  G  C?.  Assume  do  £  C->  is  not  in  the  closed  subspace  IT'  :=  {d  G  C^(5)  : 
lim*._00  | jo  —  -44’M;-,  =  0  with  G  C ^}.  Then  do  and  IT  can  be  separated  by  a  closed 
hyperplane.  I.e.  there  exists  an  M'  G  A 4^(S)  such  that  J  do dAf '  <  0  while  f  4>dM'  =  0 
for  all  d  G  IP.  In  particular  we  have  f  A$dM'  =  0  for  all  $  G  and  consequently 
M  +  QM'  G  Ma(Ux)  for  every  M  G  A4a(Lz)  and  ©  G  R1.  If  M  denotes  an  arbitrary 
fixed  element  of  .V4..i(/.x)  then  we  have 

hU^o)  =  inf  /  OodM<  inf  /  do  d(M  +  QM') 

1  M€MaJ  ©ea1  J 

=  /  do  dM  +  inf  0  •  /  do  dM'  =  —  00. 

J  ©eR1  y 

Here  we  used  that  by  assumption  /  do  dA/'  ^  0  and  that  0  cam  be  arbitrary.  This  completes 
the  proof  of  the  lemma. 

The  next  theorem  is  the  main  result  of  this  section.  Roughly  it  states  that  seeking  the 
maximal  solution  of  the  Hamilton- Jacobi-  Bellman  equation  is  the  dual  to  the  weak  problem 
formulated  in  the  previous  section.  As  under  the  current  weak  assumptions  no  smooth 


*  M  •  m  m  M  *  m  *  M  m  m  m  0  m  ^ 


solutions  to  the  Hamilton- Jacobi- Bellman  equation  need  exist,  the  precise  formulation  of 
the  duality  relationship  is  the  following.  The  value  function  (i.e.  the  minimum )  of  the 
weak  problem  is  the  upper  envelope  (i.e.  supremum)  of  the  smooth  subsolutions  of  tin 
Hamilton-Jacobi  equation. 

Theorem  1. 

\&(f,x)  :=  min{  J  IdM  :  M  £  Ma{Ux)  H  Af7,r} 

=  sup{$(f,  r)  :  A$  +  l  >  0). 

PROOF:  If  applied  to  C*  =  M±  ,  Rockafellar’s  duality  theorem  [4]  states  that 

min {h(M)  -  h2(M)  :  M  £  Ml(S)}  =  sup {h*(<?)  -  h^)  :  <f>  £  Cy(S)}  (4.7 j 

whenever  the  set  {<p  :  >  —  oc}  contains  a  finite  continuity  point  of  /?}.  But  this  con¬ 

dition  is  satisfied  since  h*  is  continuous  and  finite  on  whole  Cy  and  h^iP)  is  not  identically 
—  oo  ,  and  hence  (4.7)  holds  true. 

Substituting  the  explicit  expressions  for  h*  and  h *2  from  Lemmas  4.1  and  4.2  into  (4.7 j 
and  using  the  fact  that  {A$  :  $  £  C^}  is  dense  in  {<t> :  >  — oo}  we  obtain 

'I >(#.  x)  =  min {h^M)  -  h2(M)  :  M  £  Ml(S )} 

=  sup{$(f,x)  -  a-1  -7 (t,x)  •  ||(A$  +  l)~\\y  :  $  6  C^} 

To  conclude  the  proof  it  is  sufficient  to  show  that  for  every  $  £  C*  there  exists  a 
such  that  -f  /  >  0  and  >  $(f,x)  —  a-1  •  7 (t.x)  ■  ||(A$  4-  7)_||v 

Choose  3>~~  —  a-1  •  7||(.4$  +  /)-||7.  Then  by  Lemma  1.1  —  A*)  >  07  holds  and 

consequently,  we  have 

A*~  +  /  =  A$  +  l-  a-1  •  || (A9  +  /)- 1|7  •  A7  >  A*  +  7  +  7  ■  || (A*  +  /)“ ||7 
=  A<I>  +  /  +  7  •  sup  |(A$  -f  ,x' ,x')\  >  0. 

The  proof  of  the  theorem  is  complete. 

In  a  less  compressed  form  Theorem  1  states  that  the  weak  value  function  is  the  upper 
envelope  of  all  $  £  Cl(E )  satisfying  the  Hamilton-Jacobi  inequality. 


n  n 

x)  -f  min  {r  a(t,x.y)$ZiTJ(t,x)  +  ^6i(t,x,y)$I|.(f,x) 
y  ',>=1  »=i 

-  c(f,x,y)$(t,x)  +  l(t,  x,  y)  j  >  0. 


Recall  that  the  definition  of  includes  <$(T,  x)  =  0  whenever  T  <  -foe. 

The  fact  that  .4$]  >  A'Iq  implies  $]  <  $2  justifies  to  call  the  functions  G  C': 
satisfying  (4.8)  subsolutions  of  the  Hamilton- Jacobi  equation. 

The  results  of  the  present  paragraph  remain  valid  under  much  more  general  assumptions 
than  those  made  in  §2.  In  fact,  we  did  not  use  either  the  finite  dimensionality  of  the 
state-space  or  the  specific  properties  of  diffusion  processes.  Besides  Rockafellar’s  duality 
theorem,  our  approach  was  based  on  the  validity  of  Dynkin’s  formula,  but  not  even  the 
denseness  of  C *  was  exploited.  Since  Dynkin’s  formula  is  a  special  case  of  the  “general 
fundamental  theorem  of  calculus”  in  semigroup  theory,  all  results  of  the  present  paragraph 
can  be  generalized  to  the  case,  when  the  state  and  control  spaces  are  locally  compact 
separable  metric  spaces  and  is  substituted  by  a  linear  subset  C  of  C1(E).  Of  course, 
this  latter  change  affects  the  definition  of  M.  a  and  consequently  the  weak  problem  itself. 
But  still,  the  dual  of  this  new  “£-weak”  problem  will  be  the  problem  of  finding  the  upper 
envelope  of  all  subsolutions  in  C  of  the  Hamilton-Jacobi-Bellman  equation  involving  the 
operator  A.  The  coincidence  of  the  primal  and  dual  values  remain  preserved  too. 


5.  Equivalence  of  the  strong  and  weak  formulations 

We  prove  the  equivalence  under  the  assumption  of  a  special  approximation  property  of 
the  value  function  corresponding  to  smooth  costs.  In  Section  6  we  will  show'  that  under 
the  assumptions  of  the  present  paper  this  approximability  is  ahvavs  true. 

THEOREM  2.  Let  /  €  C^(S)  denote  an  arbitrary  smooth  “running  cost ”  and  denote  F 
the  corresponding  (strong)  value  function. 

Suppose  that  every  such  value  function  F  can  be  approximated  in  the  ||  ||7-.norm  by 

a  sequence  of  functions  F(t^  each  of  which  has  first  and  second  derivatives  essentially 
bounded  in  q-norm  and  satisfies  AF(r^  4-  /  >  0  a.e.  as  well  as  F^\T,x)  =  0  whenever 
T  <  00.  Then,  for  each  ( t,x )  G  E°  and  l  as  in  Section  2  the  weak  and  strong  formulations 
are  equivalent;  their  optimal  value  functions  coincide. 

Note  that  Theorem  2  assumes  the  approximability  of  value  functions  generated  by 
smooth  costs  and  makes  a  statement  about  the  more  general  control  problem  which  in¬ 
volves  general  continuous  or  (later)  even  only  low'er  semi  continuous  running  cost  l. 

PROOF:  Assume  that  the  statement  of  the  theorem  is  false,  there  exists  an  initial  state 
(t0- 3:0)  such  that  lI'(to,Xo)  <  tl'(to,Xo)-  This  means  that  there  exists  a  measure  M0  G 
MU  (tn,xn)\,Ms(tn.  Jn)  which  gives  rise  to  a  cost  J  1  dM0  lower  than  i/.'(f0, 1 0)  the  infimum 
over  all  costs  generated  by  controls  u  E  U,  i.e. 

J  l  dMo  <  inf  {  J  l  dMu  :  u  €  U  } .  (5.1) 

This  means  that  the  w*-continuous  linear  functional  f  l dM  on  A4±(5)  separates  an 
element  Mo  E  *M_"  from  the  w* convex-closure  of  the  set  =  { j\/u  :  it  E  U).  In  other 
words  M  U  is  strictly  larger  than  the  closure  of  MS .  If  this  is  so,  then  Mo  and  the  compact 


set  M  can  also  bo  separated  by  a  functional  f  f  dM  generated  by  a  smooth  /  £  C':t\S ). 
More  precisely,  since  smooth  functions  form  a  dense  subset  in  C7  there  must  exist  an 
/  £  C:  such  that 

J  I 

fdM0  <  inf{  J  fdMu  :  Mu  £  MS).  (0.2; 

Let  us  introduce  the  strong  value  function  F  corresponding  to  the  running  cost  / 

T 

F{t..v)  :=  ini{  J  fdM  :  M  £  AtS(*,r)}=  inf  EtiX  J  f(t,x*.ut)dt.  (0.3, 

Then,  according  to  the  assumptions  of  the  theorem,  for  every  e  >  0  there  exists  an 
F^  such  that  the  partial  derivatives  Fj€\  Fj£ \  Fz*lj  are  all  defirred  a.e.,  are  essentially 
bounded  and  for  every  y  £  Y  the  inequality 

AyE{e\t,x)  +  f(t,x,y)>  0  (5.4) 

is  satisfied  for  a.e.  ( t,x )  £  E  and  ||F^  —  F||7  <  e. 

The  generalised  Dvnkin’s  formula  (3.2)  cannot  directly  be  applied  to  (5.4)  because  F('^ 
is  not  smooth,  it  should  first  be  approximated  by  C*  functions.  The  details  of  this  approx¬ 
imation  are  presented  in  the  next  two  lemmas.  Using  them,  the  conclusion  of  the  proof  of 
Theorem  2  will  be  straightforward. 

LEMMA  5.1.  For  every  6  >  0  there  exists  an  F(r’^  £  C^(F)  such  that 


||F(£)  -F(^)|U  <  6, 

\\AF^\U<\\AF^\\1  +  6, 

and  AF(t’6)  +  /  > -6  •  i  on  [6, T  -  6}  x  R"  x  Y.  (5.5) 

PROOF :  First  we  ex  end  the  definition  of  F ^  from  [0,  T)  x  R"  to  [— T.  2F]  x  R"  by 


F(£)(-s,  x)  :=  F(t)( 0,  x)  s  £  [0,  T],  x  £  R" 

F''\T  +  s,x)  :=  Fl'\T-s,x)  (5.G) 

and  the  functions  a.b.c,  and  /  from  [0,T]  x  Rn  x  Y  to  [-T, 2F]  x  R”  x  Y  by  reflection 
over  0  and  T:  i.e.  a(-s,xy)  =  a(s,.r,y)  and  a(T  +  s,x,y)  =  a(T  -s,x,y)  if  s  £  [0,T],.r  £ 
R",y  G  V ,  and  similarly  for  6,  c  and  /. 

Note  that 


AvF(e\—s,x)  =  ~Fi'\s,x)  +  AyF('\s,x)  s  £  [0,T],.r  £  R",y  £  Y 
A"Fu\T~  s.x)  =  2Fjc\s.x)  -  AyF(€\T  -  s,x).  (5.7) 
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Moreover,  because  of  the  Lipschitz  continuity  of  we  have 

sup  \AF{C) /^\  =  I\ 

[—T,2T\xRn  x  V 

with  some  finite  number  K.  (We  reserve  the  notation  ||  •  ||7  for  sup  over  [0,T]  x  R''  x  V.) 

Let  pr(t,x )  be  a  non-negative  symmetric  C^-mollifier  (partition  of  unity)  with 
f  f  pr(cr,£)dad£  =  1  and  pr(cr,  £)  =  0  if  (erf  -f-  |f|  >  r.  If  <f>  €  C([—T,2T]  X  R")  we  define 
C  *  pr  on  [0,  T]  x  Rn  by 


(6  *  p 


JJ 


<t>(t  +  <7,  x  +  £)pr(a,  f)  dad(  if  0  <  r  <  T. 


From  the  second  relation  of  (5.6)  it  follows  that  (F^  *  pr)(T,  x)  =  0,  moreover  F^'1  *  pr  is 
infinitely  often  differentiable  on  [0,  T]  x  Rn  and  ||.F^  *pr||-)  <  I\.  Consequently  F's'  *  p,  6 
C'*(E)  for  every  0  <  r  <  T. 

Since  by  (5.4)  .4fr(i)  +  /  >  0  holds  almost  everywhere  on  [0,T]  x  Rr  x  Y,  it  follows  that 

(-4F(f))  *  pr  +  f  *  Pr  >  0  on  [r,  T  —  r]  x  R"  x  Y 

We  want  to  show  that  for  every  6  >  0  there  exists  an  r  >  0  such  that  F^e'6^  :=  F^  *  pT 
satisfies  (5.5).  Cleaily,  we  can  assume  r  <  S,  i.e.  [6,T  —  6]  C  [r, T  —  r]  and  thus  it  is 
sufficient  to  show  that 


||(>lF(e))  *  pr  —  A(F(e)  *  Pr)IU  0  and  j|/  *  pr  -  /||7  -  0. 


We  have 


-^—j[(AF(e)  *  pr-  A(F(e)  *pr)](t,x,y ) 

=  Z^J~r)  f  f  {it,  [a*>(*  +  <r,*  +  t,y)~  <*•](*,  x,  (t  +  cr,x  +  0 

n 

+  [b.(<  +  X  +  t,  y)  -  bi{t,  x,  y)]  F^\t  +  cr,  x  -f  0 

i  =  ] 

-  [c((  +  a,x  +  (,y)  -  c[t,x,y)]F('\t  +  <?,x  +  ()}pA’’-.Od,’d( 

<  E«.je)l!fi';,ll,  +  E‘o(>-)l|fi'’ll,  +  f(r)l|F<tl|i1  <o.6) 

where  a,^,6,.  c  denote  the  moduli  of  continuity  of  the  corresponding  coefficients.  Since  the 
coefficients  were  assumed  to  be  uniformly  continuous  and  the  ||  •  H-,— norms  of  Ft* .  F{z*] 
and  F(r)  are  finite  by  the  assumption  of  the  theorem,  the  right-hand  side  of  the  inequality 
tends  to  zero  as  r  — ►  0,  proving  the  lemma. 


CUCWJtfN' 
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Lemma  5.2. 


f  7  dM  <  (t2  -  tj)  ■  -y(t.x)  (0.0. 

,t:]xRn  xY 

holds  true  for  every  M  £  Ad"  (t,r)  and  0  <  ^  <  T. 

PROOF:  Denote  x(s)  :=  (<2  —  i\ )  —  /0*  l[f t ,<2] (cr)c^cr  and  let  x*  ■  [0,  T]  — ►  R1  be  a  monotonely 
decreasing  sequence  of  functions  which  are  continuously  differentiable  in  (0 ,  T),  for  which 
Xk{T)  =  0  and  such  that  Xk  \  X  and  x'k  /  — 1[<! ,t2]  =  x'  as  k  ->  oc. 

For  M  £  Ad"  (f,x)  C  Ad^(t,:r).  the  generalized  Dynkin’s  formula  (3.2)  can  be  applied 
to  the  functions  $*(<7,  £)  :=  Xk(&)  •  7(c,  £).  Using  relation 

A$*  =  7  •  A**  +  x*  •  A7  =  7  •  x*  -  7  •  c  •  X*  +  Xk  ■  A-} 
and  the  fact  that  c  ■  7,  —  Aq,  and  M  are  non-negative,  we  obtain 


(f 2  -  f  1 )  •  7'(f  1  •*•)  >  7(t,  x)  •  \(0  =  Km  7 (<,  ar)  •  Xfc(t) 

fc— *00 


=  lim  <$*(*,  2-)  =  lim  /  —A$kdAl 

*— » 00  *— *00  y 

=  fcKrn^  J  -x'k  -7  dM  +  J  XkC  ■  ~l  dM  +  J  Xk{~A~,)dM 

>  Kn^  J  ~ Xk  •  7  dM  =  J  1(<1)<2]  •  7<fA/ 


proving  the  lemma.  ' 

Conclusion  or  the  proof  of  Theorem  2: 

Since  F(f^  £  C;(F)  and  M0  £  Ad"'  C  Ad  A (t,  x),  we  can  apply  the  generalized  Dynkin's 
formula  and  obtain  by  (5.5)  and  (5.9) 


F(e’e)(t.x)  =  -  f  AF(e' 

J[0,T]x'Rn  xV 


6)  dM0 


—  [  f  dMo  -f  6  •  j 

J[6,T-6]*KnxY  J[6,T-6}xK”xY 


7  dMo 


I 


7  dMo- 


J({0,S]u[T-S,T])xKnxY 

Since  M0  £  Ad"  C  A/P,r  ,  we  have  0  <  Js~fdM0  <  T.  From  Lemma  5.2  it  follows  that 
the  integral  in  the  last  term  is  not  greater  than  26  •  x(t,  x)  =  26  ■  a  •  T  and  since  by  Lemma 
5.1  ||AF<r-*>||7  <  ||AF(e)||7  +  6  ,  we  have 


r’f)(t,x)<  J  f  dMo  +  6  •  2(1  -+■  allAF^U-y)  •  T 


(5.10) 


Choosing  first  5  then  6  sufficiently  small,  from  \\F  —  Fe,#||  <  e  +  6  and  relation  (5.10)  it 
follows 

F(t.x)  =  inf  |  J  f  d\Iu  :  Mu  €  AiS}  <  J  f  dM0 

in  contradiction  to  the  choice  (5.2)  of  /  as  separating  functional.  This  proves  the  equiva¬ 
lence  of  the  strong  and  weak  formulations. 

Remark.  Assumptions  on  the  derivatives  of  F ^  were  only  needed  to  obtain  estimate 
(5.S).  Note  that  since  F<£>  is  locally  Lipschitzian,  its  first  derivatives  exist  a.e.  and  are 
locally  bounded.  This  fact  alone  is  sufficient  to  prove  the  equivalence  of  the  strong  and 
weak  problems  provided  the  diffusion  coefficients  a,j  do  not  depend  on  t  and  x.  In  fact, 
in  this  case  the  terms  [atJ(<  -fi  cr,  1  +  £,  y)  —  cqj(<,x,y)]  are  zero  and  no  assumptions  on 
the  second  derivatives  FXi Z]  are  needed,  shortcutting  the  approximation  by  F ^  and  the 
entire  Section  G. 

COROLLARY  1.  Suppose  that  l  is  of  at  most  linear  growth,  i.e.  |/(f,x,y)|  <  -t-rj  |x|  +  r2t . 
If  the  processes  are  deterministic  (atJ  =  0)  or  the  diffusion  coefficients  are  independent  of 
time  and  space,  then  the  strong  and  weak  problems  are  equivalent. 

PROOF:  If  l  is  of  linear  growth,  then  7  can  be  chosen  to  be  r  •  (1  +  |x|  -f  t )  with  r  > 
max(r0,r1,r2).  Consequently  /  will  be  uniformly  Lipschitzian  and  so  will  be  F.  Moreover 
F  can  be  represented  as  the  sum  of  a  concave  and  of  a  smooth  function,  hence  the  first 
and  second  partial  derivatives  of  F  exist  a.e.  and  the  first  partials  are  uniformly  bounded. 
The  corollary  then  follows  from  the  previous  remark. 

The  measure  M0  introduced  in  the  proof  of  Theorem  2  could  not  be  in  the  w*  convex- 
closure  of  MS(U.  xn).  The  argument  there  in  fact  proves: 

Corollary  2.  ,MU  ( t,x )  is  the  w*  convex  closure  of  Ms(t . x). 


G.  A  Sobolev  approximation  of  the  value  function 

To  complete  the  proof  of  the  equivalence  of  the  strong  and  weak  problems  it  remains 
to  show  that  the  value  function  generated  by  a  smooth  running  cost  can  be  approximated 
by  Wl£  function  the  way  required  by  the  assumptions  of  Theorem  2.  This  kind  of  ap- 
proximabilitv  of  the  value  function  which  does  not  use  any  non-degeneracv  assumptions 
is  also  of  independent  interest  in  other  branches  of  control  theory  unrelated  to  the  strong 
and  weak  formulations.  This  section  is  devoted  to  the  proof  of  the  result. 

THEOREM  3.  Let  f  G  C^(S)  and  let  F  be  the  corresponding  value  function  defined  by 
(5.3).  Then  for  every  e  >  0  there  exists  a  function  F^  £  C1{E)  with  the  following 
properties 

(a) HF-F<'>ll,<e: 

(b)  The  partial  derivatives  Ffe\ Fr‘\ F^Xj  exist  almost  everywhere  for  every  1  <  i.  j  < 
n  and  satisfying  |||F(e^||j7  <  K(e)  with  some  constant  K(e)  where 


v  ff  <|1  _■  |l.' 


^"iiu  *  i^’i,  +  +  E  +  E  iv 


(c)  AyF^\t.x)  +  f(t.x)  >  0  for  almost  every  (t,x)  G  E,  for  every  y,  and  F(e\T,x)  =  0 
whenever  T  <  oc. 


We  denote  the  weighted  Sobolev  space  of  all  functions  satisfying  (b)  by 

The  idea  of  the  proof  is  to  extend  the  control  set  of  the  original  problem  by  one  additional 
“smoothing  control’’  giving  rise  to  an  n-dimensional  Brownian  motion.  The  value  function 
of  the  extended  problem  will  then  have  the  required  smoothness  properties  and  by  charging 
a  sufficiently  high  penalty  for  the  “smoothing”  its  domain  of  application  can  be  kept  small 
and  this  way  the  smoothed  value  function  can  be  forced  to  remain  close  to  the  original 


To  be  more  precise,  let  us  introduce  one  more  additional  control  77  so  that  the  extended 
control  set  will  be  1  'U  {77}  .  The  process  associated  with  r]  will  be  the  standard  n-dimensional 
Brownian  motion  discounted  at  the  lowest  possible  rate  Co  =  inf t,z,y  c(t,x,y)  so  that  we 
have 


£>%*W+.)  = 


/  4(' +  <>“»  I"  «  “=  <*•  *  ')• 


The  infinitesimal  operator  corresponding  to  the  exponentially  killed  Brownian  motion  is 


A"$  =  -A$-co$ 


where  A  denotes  the  Laplacian.  Recall  that  in  Lemma  2.1  inequality  (2.2)  holds  not  only 
for  the  family  of  operators  {Ay  }  but  with  possibly  different  numbers  a  and  d  also  for 
the  extended  family  {^4y}ygyu{77}-  In  particular  we  have  0  <  a 7  <  —  Avy. 

During  the  period  of  time  when  the  new  control  77  is  applied  we  charge  the  running  cost 


f{s,x,r])  :=  L  •  (~Avy){s,x)  =  L  ■  (co lf(s,x)  -  -At(s,x)) 


with  some  constant  L  to  be  determined  later.  For  simplicity  we  only  allow  77  to  be  applied 
during  at  most  one  non-random  interval  of  time.  In  other  words,  the  extended  set  U v  of 
admissible  controls  will  be  the  set  of  all  functions  of  the  form 


V(uj  s)  -  (  77  if  <1  <  -s  <  <2 

\  u(u>,s)  otherwise 


with  all  possible  choices  of  0  <  t\  <  t2  <  T  and  u  €  U.  Note  that  because  of  the  possibility 
of  killing,  the  processes  may  die  before  tx  or  t2. 

The  value  function  of  the  extended  problem  can  then  be  written  as 


CilTjWTlWtTiTrtll.i1 


vw 
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FL{t,x):  =  inf  E't  [  [/(s,  x*\  r4)ly  (ra )  +  I  •  {-A*’)  )(s,  x,)l  { (v9 )j  (h 

v€Un  Jt 

=  inf  E/  A  [  f(s,x*,ua)ds 

*<U<U<T  <Xl  Jt  *’ 

+  El%<  \f  L-  (-A"-i)(s,i^)ds  +  F((2,x  J,)]  } 

1  J  <1 

<  t<t}j£a<TEZ*{  f  f(s,x«,ua)ds  +  (0ta-tl  *  F)(/i,x"i) 

_  \i£U~  Jt 

+  L  ■  (~f  -  /3,2-zj  *  7)(fi,x”)}.  (6.1) 

Now  we  show  that  setting  the  penalty  L  high  will  keep  the  optimal  cost  FL  close  to  F. 

Proposition  6.1.  For  every  e  >  0  there  exists  an  0  <  Le  <  oc  such  that  ||F  —  F1,  ||7  <  e. 

PROOF :  FL  <  F  is  trivial  since  <bL  is  the  value  functional  of  the  extended  control  problem 
which  contains  the  original  problem  embedded,  as  t2  =  t\  is  permitted. 

To  show  F  —  FLt  <  e  ■  7  observe  that  since  F j 7  is  bounded  and  uniformly  continuous 
there  exists  a  tc  such  the  HF/7  —  flh  *  (F/7)||  <  e/2  for  all  0  <  h  <  te.  With  this  tc  let 
us  choose  Le  :=  3||/||7/(o  •  tc).  Now  let  us  consider  an  arbitrary  (f,: r)  E  E.  Since  FLt  is 
the  pointwise  infimum  in  (6.1),  we  can  find  an  e/2-optimal  triple  u,  0  <  fj  <  t2  i.e.  such 
that 

T(t,r)  :=  £,%{  jP  f{T„v.)  +  {8h  *  F)  (Fj.XjJ  4-  Lt  •  E{^  (~A^)(ws)  ds) 

<  FL'(t,x )  +  |  * 7(/,ar) 

We  use  the  notation  xa  =  1“  and  h  =  t2  —  7 1 .  Keep  in  mind  that  although 
do  depend  on  {t,x).  the  numbers  te  and  La  were  chosen  before  (t,x)  was  picked,  hence 
estimates  involving  only  te  and  Le  will  hold  for  every  (t,x)  E  E. 

With  the  quantities  just  defined  we  can  write 


F(t.  x)  —  FL‘ (t,  x)  =  [F(t,x)-F(t,x)]  +  [F(t,x)~  FL‘(t.x)] 

<  [F{t,x)  -F(t,x)]  +i{t,x)  ■  |  (C.2) 

and  it  remains  to  show  that  F(t.x)  —  F(t,x)  <  7 (t.x)  ■  ~ . 

We  increase  the  value  if  we  fix  11  for  the  initial  interval  [t,  7j )  and  allow  minimization 

only  after  t\ . 


(0.3  i 


Fit.  x)  -  F(t.x)  <  Et,x{j  f{s.xs.ua)ds  4-  F^r,-,)  j  -  F{t,x) 

=  Et,{([F  -  fa  »  F]  -  L,  •  [7  -  0k  »  •,])((„!,,)}. 

The  expression  in  the  first  bracket  under  the  expectation  sign  normalized  by  7  can  be 
estimated  at  an  arbitrary  (f1,!1)  G  E  as 


1 _ 

'r(t'.x') 


[F(t\x')-(0h*F)(t',x')} 


(<',*')  + 


(t'.x') 


< 


+  m\y 


Ph  *  ~,(t',x')\ 
l(t'.x')  ) 


Consequently  if  we  divide  the  whole  expression  under  the  expectation  in  (6.3)  by  q  we 
obtain  for  it 


W'.*')  ■■=  ^7,  •  ([r  -  0H  .  F]  -  L.  ■  [7  -  A,  •  7])((',x') 


< 


(l<  -  imi,)  • 


(3h  *  7(f,,x')\ 
7  {t',x')  )' 


(6.4) 


Now  there  are  two  possibilities:  either  h  <  te  or  h  >  te.  If  h  <  tc  then  by  the  definition 
of  te  we  have  ||F/-,  -  (3h  *  (F/7)||  <  e/2.  Since  I£  >  ||/||7/a  >  ||F||7  and  7  >  (3h  *  7, 
the  last  term  is  non-negative;  we  may  subtract  it  and  we  get  x(t',x')  <  e/2  for  arbitrary 
(t',x')eE. 

On  the  other  hand  if  h  >  te,  then  1  —  e~ah  >  1  —  tat'  thus  by  Corollary  3  to  Lemma 
2.1  and  the  choice  of  Lc  we  have 


(If  -  ||F|!- ) 


(h_n >  nfh  /,  _ 

7(t',x')  J-a( 


211/11, 


>2||  F||v 


Since  both  F/7  and  @h  *  (F / 7)  are  bounded  by  ||F||7.  from  (6.4)  we  find  that  \(l'.x')  < 
0  <  e/2  for  evert'  (t'.x1)  £  E. 

Substituting  this  result  back  in  (6.3)  we  find  by  Corollary  3  to  Lemma  2.1  that 
F(i.x)  -  F(t.x)  <  F"T7(f •  x(f3.ftl)  <  |£*e7(<i^i,)  <  5  •  7(L*)- 
This  together  with  (6.2)  gives  F  —  FL‘  <  e- 7  which  completes  the  proof  of  the  proposition. 


It  is  well-known  (cf.  e.g.  [2],  Th.  4.2)  that  under  the  conditions  of  Theorem  3  the  value 
function  FL  permits  the  decomposition  FL  =  FL+FL  where  FL  £  C-,  is  smooth,  its  partial 
derivatives  F,L .  Fzzt ,  F^  belong  to  C-,  while  Fl  £  C-,  is  concave  in  x  and  monotone  in  t. 
In  fact,  for  every  control  Ju  £  C*  and  the  infimum  of  continuously  parameterized  family  of 
C*  functions  lias  the  above  decomposition  property.  For  such  functions  the  generalization 
of  Alexandrov's  theorem  [1]  holds  true;  for  almost  every  ( t ,  x)  the  derivatives  FtL  ,FX. ,  F/.x 
exist  and  satisfy 

FL(t  4  a,x  4  £)  =  FL(t,x )  4  FtL(t,x )  •  a  4-  ^  x)£, 

+  +  °0fi +  i£i2)-  (c- <> ) 

It  is  easy  to  see  that  FL  satisfies  the  Hamilton-Jacobi-Bcllman  inequality  of  the  extended 
problem  almost  everywhere.  In  fact,  the  next  proposition  is  only  a  slight  modification  of 
known  results  (cf.  [2],  [3])  which  we  prove  here  only  because  the  easy  proof  makes  our 
exposition  self-contained. 

Proposition  6.2.  For  every  y  £  Y  u  {r?} 

AyFL(t.x)  4  f(t,x,y)  •  ly  (y)  4  L  ■  (-Ay-f)(t,x)l{v](y)  >  0  (6.6) 

for  almost  every  (t,x)  £  E. 

PROOP:  Suppose  there  exists  a  y  £  Y0  and  a  (to,x o)  £  E  from  the  noil-exceptional  set 
such  that 


A^°FL(to,x0)  4  /(<o,*o,yo)  <  -6  <  0. 

Then,  by  the  continuity  of  the  underlying  processes,  there  exists  an  So  >  0  such  that  for 
all  s  <  s0 


,-i 


EZIorL(to  +  «di)  -  F(f0,x0)  4  /(fo»*o,yo)  <  -5/2  <  0. 


Let  £  Uv  be  a  6/3-optimal  control  for  the  initial  state  (to  4  so.O  and  define 

f  y0  if  t0  <  t  <  t0  4  s 
u  (u\t)  „ 

1  u|  if  t  >t0  +s  and  x<0+,0(u>)  =  f. 

Then  this  control  is  again  in  Un  and  will  yield  the  cost 


Et°fZo  j  f{s,xa,u0(s ))  ds  <  f(t0,x0,y0) '  50  +  Eto,*0  {fL  (*°  +  s’x«o+*0)  +  6/3  J 

<  F(f0,  x0)  -  6/6  <  0 


in  contradiction  to  the  definition  of  Fl  as  the  infimum  over  all  a  £  U,v  The  proof  for 
yo  =  r]  is  the  same. 


t  jnrw  v  x  u  w  ’j*>rL 


Conclvsion  or  Tiir  Proof  or  Theorem  3: 

Let  us  choose  :=  FLl  according  to  Proposition  6.1.  Then  we  have  jjF  —  T  ’  S  -■ 
The  derivatives  F,(f\ F*f' ,  F^]z  exist  almost  everywhere  by  Alexandrov's  theoiem  and 
Proposition  G.2  shows  that  the  Hamilton-Jacobi  inequality  holds  true  for  every  y  G  T  and. 
for  almost  every  {1.x)  G  E.  The  smooth  component  F^  and  its  derivatives  are  in  C-.  by 
Krylov’s  cited  result  ([2],  Th.  4.2).  ^ 

It  remains  to  show  that  the  derivatives  of  the  concave  component  F(f^  are  essentially 
bounded  by  K{e)  •  7. 

Consider  the  first  derivative  in  an  arbitrary  direction  of  the  (t,x)- space.  By  the  con¬ 
cavity  of  Fu)  this  directional  derivative  is  monotone  along  each  line  parallel  to  the  chosen 
direction.  Suppose  that  this  (one-dimensional)  derivative  function  exceeds  K  ■  7  for  every 
K .  Then  by  its  monotonicitv  follows  that  neither  can  its  integral  function  be  bounded  by 
Ki  •  7.  But  this  contradicts  F(f)  €  C-,  which  follows  from  Proposition  6.1.  Hence  there 
must  be  a  Ki{e)  such  that  |F((e)(t,  *)|+£"_,  l-F^Ct,*)!  <  A'2(e)7(t,x)  almost  everywhere. 

As  for  the  second  derivatives,  FZi’Ij  (t,  x)  <  Ks{£)  follows  from  the  concavity  of  F  '' .  To 
show  |Fifij  |  <  A'(e)  •  7  consider  inequality  (6.6)  of  Proposition  6.2  for  y  =  77.  This  claims 
that 

F«(°  +  Fie)  -  c0F(£)  >  Lt  ■  A'*,  >Lt-  a  •  7 

where  the  last  inequality  follows  from  the  right-hand  side  of  (2.2).  Using  the  estimates 
already  obtained  for  F(e\  Ft(e\  F^TJ  we  get 

>  -2(c||F<*>||,  +  llff’ll,  +  i  £  nn.,,11,  +  L,o)  ■ 

0  =  1 

This  lower  bound  for  the  sum  £  F^]Zj  /7  together  with  the  upper  bound  for  the  individual 
summands  Fr(£  obtained  from  the  concavity  of  F(e)  implies  ||Fi^rj- |]T  <  K(e)  for  every 
1  <  Fj  <  n.  This  completes  the  proof  of  Theorem  3.  ... 


7.  Semi-continuous  costs 

In  the  previous  paragraphs,  in  particular  in  §3  and  §4,  we  assumed  the  running  cost  to 
be  continuous  I  G  Cv  Now  we  are  going  to  remove  this  assumption  and  allow  l  to  be  lower 
senn- continuous  and  of  growth  less  than  7.  More  precisely  we  denote  bv  LC-,  the  set  of  all 
functions  l  satisfying  * 

(i)  I  is  lower  semi-continuous 
(n)  sup(^j,)65  |/(f, y)\/y(0  <  00 
(iii)  limsup,^^  1(£,y)/y(£)  =  0. 
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Such  function?  can  be  represented  as  upper  envelopes  of  continuous  functions  /  =  sup {  f  : 
/  €  C'-> ,  /  <  /}  or  even  as  limits  of  non-decreasing  sequences  of  C-,  functions. 

The  aim  of  the  present  paragraph  is  to  show  that  all  results  proved  for  continuous  / 
in  the  preceding  paragraphs  remain  true  for  control  problems  with  lower  semi-continu< 
cost  functions  /  £  £CV  The  key  tool  in  approximating  lower  semi-continuous  cost-  by 
continuous  ones  will  be  the  following  min- max  type  argument. 

PROPOSITION  7.1.  Suppose  l  £  LC-,  and  let  K  denote  an  arbitrary  ir* -compact  subset  o: 
Ml(S).  Then 

inf  I  1  dp  =  inf  sup  f dp  =  sup  inf  /  f  dp.  (7.1 

J  fieK  f<tjec^  J  f<tjecy>i€KJ 


PROOF:  Note  first,  that  every  1  £  LC-,  defines  a  convex,  lower  u:*-semi-continuous  func¬ 
tional  on  .Vd^..  hence  all  infiina  in  (7.1)  are  attained  for  some  elements  of  the  ic*-compac: 
set  K. 

The  monotone  convergence  theorem  and  the  obvious  inequality  inf  sup  >  sup  inf  yield 

Jo  :=  inf  /  l  dp  =  inf  sup  /  f  dp  >  sup  inf  /  f  dp. 

Kf<U€cJ  ~  f<i,f€C\MeKj  J 

Let  pf  denote  the  measure,  for  which  f  f  dpf  =  inf^g*:  f  f  dp.  To  prove  the  proposition 
it  is  sufficient  to  show  the  existence  of  a  p*  €  K  for  which 


sup  I  /dp*  >  f  l dp* 
r<i,/ec^  J  J 


holds  true. 

Let  /*  denote  a  monotone  non-decreasing  sequence  of  continuous  function?  with  fk  £ 
C\(S)  and  fk  y  l  as  k  — >  oc.  Since  K  is  sequentially  compact,  one  can  select  a  subsequence 
k,  such  that  p ,  :=  p'k>  converge  weakly*  to  a  limit  p*  £  K  as  i  —*  oc. 

Let  us  consider  the  following  array  of  reals 


bj)  :=  J  fk,  dpj  ij  =  1.2.... 


If  i'  <  i  then  I(i'.j)  <  I(i,j)  because  the  sequence  /*,  is  monotone  non-decreasing.  The 
measure  p,  is  by  definition  minimizing  f  fk,  dp  and  as  /*,.  <  /,  we  have 

Ab  0  =  J  fki  dpt  =  mf  J  fk,  dp  <  jnf  J  l  dp  =  I0. 

Consequently  all  elements  J(t,  j)  with  i  <  j  (i.e.  above  the  diagonal)  are  uniformly  bounded 
by  Jo-  From  the  monotonicity  of  the  sequence  /*,.  it  follows  that  the  diagonal  sequence 
J(i,i)  is  monotone  non-decreasing  and  so  I x  :=  lim,—^  I(i.i)  <  Jo  exists. 

Since  fk,  is  continuous  and  p*  =  u'*-lim//;,  it  follows  that  the  sequence  J(t,  j)  converges 
for  any  fixed  i  to  a  limit  I(i.  oc)  =  f  fk,  dp *  as  j  — ►  oo.  From  J(?,j)  <  I(j,j)  for  i  <  j  it 
follows 


••  -  -■  - 


i  M  *  *  w  V  •-  *.  V  V 


/(?'.  oc)  =  lim  /(?,  j)  <  lim  I(j.j)-  1^. 

]  —  OC  J  —  oc 

Recall  that  the  sequence  /*.-  was  chosen  such  a  way  that  /*..  S  1.  Consequently  the  monotone 
convergence  theorem  yields 

J(/,oc)=  ffkidpm/'  jldp*<Ioc  =  sup  /  /  dpf . 

J  J  f<ijec ,  J 

In  other  words  //*  satisfies  (7.2)  and  the  proof  is  complete. 

THEOREM  4.  Suppose  l  €  LC-y.  Then  the  (strong)  value  function  of  the  stochastic  con¬ 
trol  problem  formulated  in  §§1-2  is  the  upper  envelope  of  the  smooth  subsolutions  of  the 
Hamilton- Jacobi-Bcllman  equation,  i.e. 

ty(t,x)  =  sup{<$(t,x)  :  £  C^,  +  /  >  0}.  (7.3) 

PROOF:  It  was  shown  in  §§5-6  that  AdVV  is  the  closed  convex  hull  of  AdS.  Since  f  IdM  is 
a  convex,  lower  tr*-semi-continuous  functional  on  M±  whenever  /  6  LCy  it  follows  that  its 
infimum  over  Ads  is  the  same  as  its  minimum  attained  in  Ad "  .  Consequently  the  strong 
and  weak  value  functions  coincide  even  if  l  is  only  lower  semi-continuous  /  £  ZCV 

We  know  from  Theorem  1  that  the  value  function  permits  representation  (7.3)  if  /  is 
continuous  (/  £  C'-. ).  Proposition  7.1  can  be  applied  to  /  £  LCy  and  AT  =  Ad"  as  Ad"  is 
a  u>*-compact  set  and  we  obtain 

4'(Cx)  =  inf  [  IdM  =  sup  inf  /  /  dM 

f<t,fec^  J 

=  sup  sup{$(£,x)  :  $  £  C^(E),A$  -f  /  >  0) 

/<h/€C% 

<  sup{$(Zx)  :  $  £  C*,  A$  +  /  >  0}. 

The  opposite  inequality  is  immediate,  since  for  every  $  £  with  +  /  >  0  and  for 
every  M  £  Ad"  =  .\d"  fl  M.a  Dynkin’s  formula  yields 


t,x)  =  j  {-A$)dM  <  J  h 


Taking  infimum  over  M  £  gives  $(t,x)  <  infM€>vlw  J  l  dM  =  V-(C  x)  for  every 

$  £  C*  with  .44’  +  l  >  0  which  completes  the  proof  of  the  Theorem. 


8.  Inclusion  of  terminal  penalties 


In  this  final  paragraph  we  explain  how  to  extend  the  main  results  of  the  paper  to 
problems  where  the  cost  function  includes  also  an  additional  terminal  penalty,  i.e.  where 
the  objective  is  to  minimize  the  functional 

=  l(t,xumt)dt  +  L(x  r)}  (T<oc)  (8.1) 

over  all  controls  u  £  U.  Here  both  l  and  L  are  lower  semi-continuous  functions  of  growth 
less  than  -)  at  infinity.  This  will  extend  the  scope  of  the  results  to  include  problems  like  the 
maximization  of  the  hitting  probability  of  a  closed  target  set  or  the  fixed  end-point  problem 
of  deterministic  control  theory  which  were  beyond  the  reaches  of  the  other  approaches  to 
the  Hamilton- Jacobi  theory. 

The  key  to  the  extension  is  to  consider  a  more  elaborate  state  space  S  which  is  composed 
of  So,  the  compactification  of  the  “interior"  of  the  state-space,  and  of  Sa,  the  compactified 
“terminal  boundary,’-  as  two  separate  components.  More  precisely  let  5  denote  the  com¬ 
pact  metric  space  which  consists  of  the  two  isolated  subsets  So  :=  E  x  Y  and  Sq  :=  R" . 
Note  that  So  is  the  same  space  which  was  denoted  by  S  in  §2. 

Every  continuous  function  £  C7(S)  will  then  correspond  to  the  pair  $js0  £  C7(So) 
and  $|s8  £  C7(Sa)  where  C7(So)  is  C7(S)  of  §2  and 

C7(S9)  :=  {d  £  C(Rn) :  sup  \4>(x)\/'y(T,x)  <  oc 

r  GRn 

and 


lim  |<Hx)|/l(T,*)  =  0}. 

khoc 

LC1(S )  will  denote  the  set  of  all  lower  semi-continuous  functions  on  S,  i.e.  those  which  can 
be  represented  as  upper  envelopes  of  families  of  C7(S)  functions.  The  dual  space  to  C7(S) 
will  be  the  set  M±fS)  of  all  pairs  of  measures  M  =  (Mo, Mg)  with  il/0  £  AfJ(S0),  Mq  £ 
vW^dSa)  provided  with  the  norm  ||M||7  =  /  ^(t' ,  x' )\Mo(dt' ,dx' ,dy)\  +  f  -y(T,  x)|A/a(dj)j. 
The  set  of  all  non-negative  measures  M  £  M\(S)  with  ||Jt/||7  <  T  <  -|-oc  will  be  denoted 
by  .Mi,r(S). 

Observe  that  the  function 


f  J(F,£,y)  if  a  =  (r,t,y)  £  So 
\  L(x)  if  a  =  x  £  Sd 


is  in  LC7(Sj.  The  measure  Mv  defined  on  the  Borel  sets  B  of  S  by 


r  M"(B)  of  (2.1)  if  B  C  So 

l  P*T(xi  £  B)  if  5  C  Sa 


is  in  ,M')'1+r(Sj  C  .\d4(S).  With  this  notation  the  (strong)  optimal  control  problem  with 
both  running  and  terminal  costs  can  be  formulated  as  follows 


Minimize 


J  l  dMu  over  all  u  €  U.  (S.2< 

Let  C^(E)  denote  the  set  of  all  twice  continuously  differentiable  functions  $  defined  on 
[0,X)  X  R''  for  which  4>,  are  all  in  C-l(E)(i,j  =  1,2,...  ,n).  The  differena 

to  the  definition  of  C^(E)  in  §2  is  that  now  we  do  not  require  functions  to  vanish  on  the 

exit  boundary  [X]  x  Rn  for  X  <  oc.  Recall  that  for  every  $  G  C*(E)  Dylan’s  formula 


E“T$(T,xT)-$(t,x)  =  Et%  fT  Au‘  $(s,Tj)  ds  (S.3) 

Jo 

holds  true.  If  we  introduce  the  operator  A  :  C^(E)  — >  C^iS)  by 

7-,  \  /  A*$(t,x)  if  o  =  (t,x,y)  G  So 

then  with  the  above  notation  Dynkin’s  formula  can  be  written  in  the  more  compact  form 


-$(/,x)  =  J~A*dMu.  (S.4 ) 

The  weak  problem  corresponding  to  (8.2)  can  be  formulated  as 

Minimize  JldM  overall  M  G  A47,1+T(5)  Pi  M.  A(t ,  x) 

with  MA(i.x )  :=  {M  G  Ml(S)  :  for  which  (8.4)  holds  G  C*(E)}. 

All  the  expositions  of  §§3-7  can  be  repeated  word  by  word  for  this  extended  notation 
and  we  obtain 


THEOREM  5.  The  dual  to  the  problem  (8.2)  is  to  find  the  supremum  of  all  smooth  subso¬ 
lutions  of  the  Hamilton-  Jacobi-Bellman  equation,  and  we  have 


'i'it.  .r) 


T 

jew  a)ds  +  I(xr)} 

sup  {$((,1)  over  all  €  C^(E)  satisfying 

inf  Ay$(r,0  +  /(r,C,y)  >  0  if  0  <  r  <  X, £  €  R" 

y€> 

and  $(X,e)<X(0  £eR"}. 


References 


1.  A.  D.  Aleksandrov.  Almost  everywhere  existence  of  second  derivatives  of  a  convex  function  and 
related  properties  of  convex  surfaces,  Sci.  Notes  Leningrad  State  Univ.  37(6)  (1939).  3-35. 

2.  N.  V.  Krylov,  Some  new  results  from  the  theory  of  controlled  diffusion  processes,  Mat.  Sbornik  109 
(1979).  1-46—16-1 . 

3.  P.L.  Lions.  Optimal  control  of  diffusion  processes  and  Hamilton- Jacobi- Bellman  equation  J:  The 
dynamic  programming  principle  and  applications,  Comm,  in  Partial  Differential  Eqns.  8  (19s3j. 
1101-1174. 

4.  R.  T.  Rockafellar,  Extension  of  Fenchel’s  duality  theorem  for  convex  functions,  Duke  Math.  J.  33 
(1966),  81-89. 

5.  D.  Vermes,  Optimal  control  of  piecewise  deterministic  Markov  processes ,  Stochastics  14  (1985). 
165-208. 

6.  R.  B.  Vinter  and  R.  M.  Lewis,  The  equivalence  of  strong  and  weak  formulations  for  certain  problems 
in  optimal  control ,  SIAM  J.  Control  16  (1978),  546-570. 

7.  R  B  Vinter  and  R.  M.  Lewis.  A  necessary  and  sufficient  condition  for  optimality  of  dynamic 
programming  type,  making  no  a  priori  assumptions  on  the  controls.  SIAM  J.  Control  16  (1978). 
571-583. 


